Regression testing is a way of testing software where the test verifies that the functionality of the software that worked previously continues to work. One form that regression testing takes is for the software to be executed and output collected from the software. The output is saved as the regression control. At a later point (often repeated at regular intervals), the software is executed again and further output collected. This second output is then compared to the regression control. If the comparison reveals differences between the second output and the regression control, then a regression has been found. Subsequently, the tester has to manually determine whether that difference was due to an acceptable change in the software or whether an error was introduced.
Complexities arise, however, when the output has elements within it that change in an acceptable manner. For example, the software might include its version number and release date within the output. Whenever a new version of the software is released, this information is expected to change within the output. In this scenario, the regression control may be updated to contain the new version and release date information when the new version is released, so that the regression test does not fail. If a new version of the software is not released very often, this method may work well to keep the regression test working properly.
In another example, the software might include within the output data, the date and time at which the output was generated. This is an example where a portion of the output (date and time) is expected to always be different. The method used to update the regression control in the previous example will not work in this case, because the regression control is out of date as soon as it is created due to the fact that the output changes every time the software is executed.
In these kinds of cases, one of two things can happen. Either a regression test will not be created using this output, or the date and time will be somehow manually marked as a portion of the control that is allowed to be different when the comparison is made. Currently the process of marking the difference as allowable is a manual process, whereby a human operator will look at the differences found, determine which ones are allowable, and then somehow modify the comparison process so that the differences are not determined to be errors when the regression test is run. This process can become a lot of work for the operator, if there are many differences that need to be marked in this way within a single regression test or across other similar regression tests that the operator may be using.
Therefore, there is a need for a more efficient and effective approach to conduct regression testing and generate regression test controls.